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I.   Introduction 

In  the  past,  management  scientists  have  mainly  focused  their  attention 
on  the  design  of  decision  systems  aimed  at  the  solution  of  programmable 
and  recurring  problems.   Such  areas  as  inventory  control  and  refinery 
scheduling  can  now  be  managed  almost  automatically  by  computer  driven 
mathematical  models.   Although  such  models  may  require  extensive  as 
well  as  complicated  mathematical  manipulations,  yet  in  their  normal 
use  they  are  rather  simple  in  that  the  fundamental  relationships  en- 
compassed by  these  models  are  well  prescribed.   Martin  Starr  (Starr  1966) 
refers  to  problem  situations  which  can  be  depicted  by  such  deterministic 
planning  and  control  models  as  "fully-constrained,"   because  their 
associated  environments,  although  these  may  be  evolving,  they  are  con- 
sidered to  be  perfectly  predictable  and  all  sequences  of  events  are 
known  with  certainty. 

Next  in  terms  of  complication  come  planning  models  which  are 
probabilistic  in  nature  either  in  their  inputs  (both  data  and  assumptions), 

or  in  the  fundamental  mathematical  relationships  among  the  variables  in- 

2 
corporated  into  these  models.    These  planning  models  which  one  may 

classify  as  partially  constrained  (Starr  1966),  lead  to  tentative  con- 
sequences requiring  the  value  judgment  of  the  decision  maker  before  a 


We  will  use  Starr's,  8,  terminology  for  classification  of  planning  models. 

2 

As  the  reader  may  have  already  observed,  we  are  classifying  these  models 

on  the  basis  of  the  structure  chosen  by  the  decision  maker  in  his  effort  to 
choose  a  course  of  action  and  not  on  the  basis  of  how  these  decision  situa- 
tions could  have  been  structured.  Obviously  fully  constrained  models  could 
be  set  up  as  probabilistic  models.  With  the  exception  of  pointing  this  out 
we  shall  not  delve  into  the  question  of  the  factors  affecting  choice  of 
models,  nor  in  the  evaluation  of  the  degree  of  comprehensiveness  of  such. 
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choice  is  made.   Finally,  in  extreme  cases,  planning  problems  may  be  de- 
picted by  threshold-constrained  systems  (Starr,  8)  in  which  the  sequence 
of  events  is  speculative,  the  environment  must  be  forecasted,  and  some 
potential  outcomes  may  be  catastrophic. 

The  problem  used  in  our  experiments  was  of  the  "partially  constrained"* 
type.   We  wanted  to  find  out  how  executives  could  deal  with  capital  invest- 
ment and  competitive  pricing  decisions  under  conditions  of  uncertainty. 

3 
Many  writers  have  pointed  out  the  value  of  formal  planning   and  also 

stressed  the  necessity  of  using  structured  situations  as  a  stepping  stone 

to  higher  level  (unstructured)  planning.   Of  course  this  is  easier  said 

than  done,  and  the  average  manager  partly  because  of  necessity  but  mainly 

for  escape  finds  himself  spending  proportionately  much  more  time  on 

operational  than  on  planning  problems.   This  relative  aversion  toward 

planning  is  due  both  to  psychological  as  well  as  methodological  reasons. 

No  doubt  planning  enforces  self  discipline,  requires  persistent  effort, 

provides  standards  which  can  be  potentially  used  by  superiors  for  control 

and  accountability,  exposes  errors  and  as  such  decreases  privacy,  enforces 

integration  and  cooperation  across  organizational  functions  and  activities, 

with  all  their  human-behavior  consequences;  and  finally  demands  a  resolution 

of  the  inherent  conflict  between  the  specific  plan  as  a  secure  basis  for 

implementing  action  and  the  plan  as  a  temporary  mechanism  for  measuring 

deviations,  learning  from  experience  and  then  updating  the  underlying 

planning  model. 


3 
Among  others,  see  Ackoff,  R.  L.,  1,  Ansoff,  H.  I.,  2,  Starr,  M.  K. ,  8, 

and  Zannetos,  Z.  S.,  10. 
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What  we  are  saying  here  is  that  "security"  and  shielding  by  the  plan 

is  only  temporary  and  the  manager  cannot  survive  with  either  chaos  or 

complete  regimentation. 


Zannetos  (10  p.  13)  has  identified  three  reasons  why  planning 
problems  are  more  difficult  than  operational  problems.   They  are: 

1.  Absence  of  structure 

2.  Absence  of  many  factual  elements 

3.  Presence  of  uncontrolled  environmental  conditions 

Realizing  the  complexity  of  planning  problems  and  the  difficulties 
associated  with  designing  useful  computerized  decision  systems  to  aid 
the  decision  maker  in  solving  such  problems,  we  embarked  upon  a  research 
project  which  called  for  the  design,  implementation  and  experimental 
use  of  a  computerized  planning  model.   The  risk  analysis  technique  (Hertz,  4) 
was  chosen  as  the  basis  of  the  model  because  it  incorporated  (1)  a  facility 
for  structuring  problems,  (2)  a  methodology  for  utilizing  subjective  esti- 
mates of  elements  for  which  no  factual  information  exists  and  (3)  a  tech- 
nique for  introducing  risk  and  uncertainty  through  the  use  of  subjective 
probability  distributions.   While  the  risk  analysis  technique  does  not 
eliminate  the  three  major  reasons  why  planning  problems  are  difficult,  it 
does  recognize  them  and  operates  on  them,  and  thus  mitigates  this  onerous 
managerial  task. 

It  was  decided  from  the  outset  to  implement  the  risk  analysis  model 
on  an  interactive  graphical  display  computer  terminal.   This  decision  was 
made  for  the  following  reasons: 

1)  Interaction  was  necessary  since  the  system  operates  on 
subjective  inputs  provided  by  the  user. 

2)  Since  planning  decisions  are  partly  based  on  value  judgments 
and  subjective  inputs,  the  user  must  have  the  capability  to 
ask  "what  if"  type  of  questions  and  receive  answers  in  real 
time.   In  this  respect  we  distinguish  between  two  classes  of 
situations.   If  the  "what  if"  question  involves  changing  the 
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values  of  some  inputs  but  does  not  involve  changing  basic 
underlying  assumptions,  it  is  defined  as  sensitivity  testing. 
If  it  involves  changing  most,  or  all,  of  the  values  of  the 
inputs  and  also  changing  the  basic  underlying  assumptions, 
then  it  is  defined  as  a  new  alternative.   These  two  uses  of 
the  system  will  be  discussed  in  more  detail  later. 
3)   Since  the  risk  analysis  techniques  utilizes  probability  distri- 
butions as  inputs  and  outputs,  it  was  felt  that  graphical 
representation  of  the  distributions  would  be  more  meaningful 
and  useful  than  would  tabular  or  parametric  representation. 

The  end  product  of  the  systems  development  effort  was  a  software  package 
called  the  Interactive  Risk  Analysis  Model  (IGRAM)  System  which  is  available 
on  the  Compatible  Time  Shared  System  (CTSS)  at  MIT's  Computation  Center. 

After  the  IGRAM  System  was  developed  during  the  early  part  of  1970, 
two  controlled  experiments  were  conducted  which  attempted  to  measure  the 
impact  the  IGRAM  System  had  on  a  decision-making  process.   The  next  portion 
of  this  paper  is  devoted  to  a  discussion  of  those  experiments. 


II.   The  Experiments 
1)   Introduction 

A  review  of  the  available  literature  on  planning  reveals  that  it 
mostly  centers  around  the  need  for  planning  models  and  to  a  much  lesser 
extent  on  the  implementation  of  models.   There  is,  however,  a  dirth  of 
literature  on  the  observed  impact  these  models  have  had  on  the  decision 
making  process.   In  an  effort  to  fill  this  deficiency  in  the  literature, 


For  a  detailed  discussion  of  the  design  and  implementation  of  the  IGRAM 
System,  the  reader  is  referred  to  Beville,  J.,  et.al.,  3. 
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it  was  decided  that  controlled  experiments  should  be  conducted  using  the 
IGRAM  System.   Our  objective  was  to  gather  data  from  experiments  which 
could  be  used  in  analyzing  the  impact  of  the  system  on  decision  making 
(both  the  results  and  the  process)  and  also  in  gaining  insights  for  the 
design  of  future  man-machine  systems. 

2)   The  Experimental  Design 

Two  sets  of  subjects  were  available  (business  executives),  so 
the  experiment  was  replicated  twice.   The  experimental  design  was  com- 
pletely randomized  with  two  treatments,  the  latter  being:   (a)  solving 
a  complex  pricing  problem  using  manual  solution  techniques  and  (b)  solving 
the  same  problem  with  the  aid  of  the  IGRAM  System.   In  each  replication 
of  the  experiment  the  subjects  were  assigned  to  two-man  teams  at  random. 
Members  of  the  odd  numbered  teams  solved  the  problem  with  manual  methods 
first  and  then  solved  it  with  the  aid  of  the  computerized  IGRAM  System. 
Members  of  the  even  numbered  teams  solved  the  problem  with  the  aid  of 
the  IGRAM  System  first  and  then  solved  it  with  manual  methods.   This 
ordering  scheme  was  intended  to  neutralize  the  learning  effect  of 
having  to  solve  the  problem  twice.   Figure  1  illustrates  the  sequence 
of  steps  taken  by  the  teams  during  the  conduct  of  the  experiment.   Copies 
of  the  referenced  questionnaires  are  contained  in  Appendix  A. 
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Figure  1 

Experimental  Procedure 
Order  of  Events 

All 
Odd  Numbered  Teams  Teams  Even  Numbered  Teams 


Answer  Questionnaire 

#1. 

Receive  Copy  of  Case 

Receive  Hertz'-'  Article 

Solve  Case  Manually 
Answer  Questionnaire  #2 


Receive  Copies  of  IGRAM 
Users'  Manual 

Briefing  on  and  Demonstration 
of  IGRAM  System. 


Solve  problem  with  IGRAM 
System 

Answer  Questionnaire  #3  Answer  Questionnaire  #4 

Solve  Case  Manually 
Answer  Questionnaire  #5 


Class  Discussion  of 
Problem  and  IGRAM 
System 


See  Hertz,  D.  B.,  (4), 


III.      The   Subjects 

a)  Senior  Executives 

The  participants  in  the  first  replication  of  the  experiment 
were  twenty  businessmen  holding  high  level  positions  within  their 
organizations  and  who  had  been  attending  the  Spring  1970  session  of 
the  Senior  Executive  Program,  at  the  Sloan  School  of  Management  at 
M.  I.T..   The  program  is  nine  weeks  long  and  is  designed  to  update  the 
senior  executives'  knowledge  of  modern  management  techniques  and  expose 
the  participants  to  present  research  and  future  trends  in  the  field  of 
management.   The  executives  were  in  their  seventh  week  of  the  nine  week 
program  when  the  experiment  started.   During  the  program,  the  executives 
were  housed  in  Endicott  House,  a  suburban  mansion  belonging  to  M.I.T.. 
With  this  arrangement,  the  participants  had  unlimited  opportunities 
for  exchanging  ideas.   On  the  whole,  the  Senior  Executives  were  in- 
terested in  the  experiment  and  enthusiastic  about  their  participation, 
which  incidentally  was  voluntary. 

b)  The  Greater  Boston  Executives 

The  participants  in  the  second  replication  of  the  experiment 
were  twenty-five  businessmen  from  the  Greater  Boston  area  who  had  been 
selected  by  their  firms  to  participate  in  the  Spring  1970  session  of  the 
Greater  Boston  Executive  Program.   The  program  is  conducted  by  the  Sloan 
School  of  Management  and  lasts  fifteen  weeks.   The  participants  attend 
classes  only  during  each  Friday  of  the  fifteen  weeks,  and  perform  their 
regular  duties  at  their  firms  during  the  other  four  days  of  the  work 
week.   The  four  class  periods  each  Friday  are  designed  to  update  the 
executives'  knowledge  in  the  areas  of  economics,  managerial  planning, 
information  and  controls,  labor  economics  and  social  responsibility. 


This  program  is  not  as  extensive  in  its  coverage  as  the  Senior  Executive  Program. 
Furthermore,  these  executives  generally  hold  positions  of  lesser  responsi- 
bility than  do  the  Senior  Executives,  are  not  as  mature  managerially,  are 
younger  in  age  and  comparatively  less  well  educated  than  the  Senior 
Executives.   The  Greater  Boston  Executives  were  in  their  sixth  week  of 
the  program  when  the  experiment  started. 

Unlike  the  Senior  Executives  (SE) ,  the  Greater  Boston  Executives  (GBE) 
lived  at  home  during  the  program.   Practically  the  only  contact  they  had 
with  each  other  was  during  the  Friday  classes  and  during  activities 
associated  with  the  experiment. 


IV.   Expected  Impacts 

If  we  use  for  reference  the  phases  of  the  decision  making  process  as 
postulated  by  Simon  --intelligence,  design,  and  choice--the  IGRAM  System 
was  expected  to  affect  only  "design"  and  "choice,"  the  intelligence,  i.e., 
problem  definition,  having  been  taken  care  of  by  the  material  distributed 
to  the  subjects.   In  particular  we  were  hoping  that  the  system  would  help 
the  user  in  the  "design"  phase  by: 

(1)  Facilitating  the  structuring  of  the  alternative  courses 
of  action 

(2)  Bringing  to  bear  on  the  problem  the  decision-maker's  subjec- 
tive estimates  of  the  values  of  uncertain  variables. 

(3)  Performing  the  complex  computations  required  to  accomplish 
(2)  above. 

(4)  Facilitating  sensitivity  testing. 


See  Simon,  H.  A.,  7.   Other  writers  on  the  subject  (especially  Ackoff  1. 
Starr  8,  and  Zannetos  10)  tend  to  look  at  these  phases  as  aspects  of  the 
planning  process. 


As  for  "choice"  activity,  the  system  was  not  so  much  intended  to 
make  the  choice,  but  hopefully  aid  the  decision-maker  in  evaluating  more 
effectively  the  alternatives  generated  by: 

(1)  Providing  more  information  than  do  manual  methods. 

The  system  accomplishes  this  mainly  by  providing  a  graph 
of  the  probabilistic  distribution  of  the  net  present 
values  rather  than  just  the  expected  value  of  such  which 
is  normally  provided  by  manual  methods. 

(2)  Displaying  information  in  a  format  which  is  easy  to  under- 
stand.  The  system  for  example  provides  graphs  rather  than 
tables  of  values  or  mathematical  descriptions  of  the  various 
probability  curves. 

(3)  Allowing  the  decision-maker  to  comprehend  the  impact  which 
his  subjective  estimates  of  risk  and  uncertainty  have  on 
computed  expected  outcomes. 

(4)  Making  it  easy  for  the  subjects  to  change  the  values  of  the 
planning  and  decision  variables  and  thus  easily  evaluate 
and  choose  among  alternatives. 


V.   Hypotheses  to  be  Tested 

The  hypotheses  which  we  wanted  to  test  were  mainly  as  follows: 

1)   The  subjects  will  tend  to  examine  more  alternative  courses  of 

action  when  using  the  computer  than  they  will  when  solving  the  case  manually, 
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Morton's  work  (Morton,  M.S.S.  5)  tends  to  support  this  hypothesis  which 
seems  to  be  rather  widely  accepted. 
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2)   The  subjects  will  tend  to  have  more  confidence  in  their  computer- 
aided  decision  than  in  their  manual  decision.   This  proposition  appears 
a  priori  to  be  reasonable  for  the  following  reasons: 

(a)  The  system  allows  the  subjects  to  bring  to  bear  all 
of  their  knowledge--objective,  subjective  and  risk 
estimates--on  the  evaluation  of  the  future  consequences 
of  each  alternative  course  of  action.   Thus  the  results 
will  tend  to  appear  to  them  as  more  dependable  and 
"scientific ." 

(b)  The  system  gives  the  subjects  better  knowledge  of  the 
structure  of  the  problem,  and  through  sensitivity  testing, 
an  idea  of  the  relative  impact  exerted  by  the  key  variables 
on  expected  outcomes.   They  will  feel  therefore  that  they 
have  a  firmer  grasp  of  the  problem  and  that  the  computer- 
aided  decision  rests  on  a  firmer  foundation. 

(c)  The  system  carries  out  the  calculations  and  gives  the 
decision-maker  additional  quantitative  information  concerning 
the  degree  of  risk  and  uncertainty  associated  with  a  proposed 
course  of  action.   The  additional  information  in  this  case 
consists  of  the  graphs  of  the  various  terminal  probability 
distributions,  which  is  normally  not  available  under  manual 
methods . 

(d)  The  system  displays  probabilistic  information  in  formats, 
namely  graphs,  which  are  easier  to  understand  than  alternative 
formats,  such  as  tables  or  listings  of  distribution  parameters, 
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(e)   Psychologically  the  subjects  may  feel  Insecure  and  tend 
to  accept  the  results  of  manipulations  they  do  not  very 
well  understand.   Furthermore,  the  feeling  that  systems 
are  designed  by  "experts"  who  undoubtedly  store  in  the 
computer  the  best  decision-making  models  known  to  them 
tends  to  encourage  too  much  faith  in  the  results  of  such 
models. 
3)   The  decisions  made  with  the  aid  of  the  computer,  that  is,  the 

courses  of  action  recommended,  will  differ  from  these  suggested  by  manual 

methods.   This  is  felt  to  be  true  because: 

(a)  The  system  provides  the  subjects  with  more  information 
concerning  risk  and  uncertainty  than  they  will  be  able  to 

Q 

generate  by  using  manual  methods. 

(b)  When  using  the  computerized  system,  the  subjects  will  tend 
to  rank  the  risk  and  uncertainty  inherent  in  an  alternative 
higher  as  a  factor  influencing  their  decision  than  they  will 
when  using  manual  methods.   This  appears  logical  since 
measures  of  risk  even  in  their  simplest  form,  e.g.,  variance, 
and  general  shape  of  the  distribution  of  net  present  values, 
will  be  nearly  impossible  for  the  subjects  to  generate  by 
using  manual  methods.   They,  therefore,  will  not  tend  to 
appreciate  the  significance  of  risk  in  decision  making.   If 
this  hypothesis  is  proven  then  certain  aspects  of  the 
educational  value  of  the  computerized  system  are  proven  also. 


g 

The  underlying  assumption  here  is  that  the  executives  are  able  to  comprehend 
and  process  this  information.   Schroder,  H.  M.  ,  et.al.,  6,  have  found  that 
beyond  a  certain  point  of  environmental  complexity  people  tend  to  process  less 
information.   In  our  case  the  structure  provided  by  the  system  reduces  the 
complexity  so  we  do  not  believe  that  we  will  be  reaching  beyond  the  maximum 
information  processing  point. 
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VI.   Measurement  Instruments 

Three  types  of  instruments  were  used  to  collect  data  during  the 
experiment.   They  were: 

1)  A  set  of  five  questionnaires. 

2)  Written  solutions  to  the  case  which  were  handed  in  by  the  teams, 
one  for  manual  solution  and  one  for  the  computer-aided  solution. 

3)  An  informal  debriefing  of  each  set  of  subjects  in  order  to 
obtain  their  reactions  to  the  IGRAM  System  in  particular  and  to 
the  use  of  planning  models  in  general.   In  addition,  the  research 
assistants  who  observed  the  subjects  during  their  computer  terminal 
sessions  gathered  valuable  informal  data. 


VII.   The  Problem 

The  problem  which  each  subject  had  to  solve  twice  and  which  was 
implemented  on  the  IGRAM  System  was  a  capital  investment  case  study.   IVo 
basic  alternative  courses  of  action  were  open  --  1)  discontinue  producing 
a  textile  product  which  was  a  part  of  the  firm's  full  line  of  textiles, 
sell  off  the  associated  equipment  and  inventory,  and  collect  the  associated 
accounts  receivable;  or   2)  continue  producing  the  product  for  the  next 
four  years  (the  equipment  would  be  worn  out  at  that  time)  and,  given  that 
decision,  further  decide  upon  the  best  pricing  strategy  to  follow. 

The  problem  was  implemented  on  the  IGRAM  System  in  the  form  of  a 
problem  tree,  as  shown  in  Figure  2,   The  "Stay  in  Business"  alternatives 
1  and  2  represent  two  different  pricing  strategies  over  the  four  year  time 
horizon.   Such  elements  as  Industry  Volume  and  Market  Share  would  be  adjusted 
in  each  case  to  correspond  with  the  particular  pricing  strategy  chosen.   For 
instance,  a  high  pricing  strategy  might  call  for  a  smaller  industry  volume 
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and  a  smaller  market  share  than  would  a  low  pricing  strategy.   All  inputs 

to  the  elements  on  the  end  points  of  the  tree  are  in  the  form  of  cumulative 

9 
subjective  probability  distributions. 


VIII.   Analysis  of  the  Experimental  Data  . 

We  will  now  analyze  some  of  the  data  which  were  collected  on  the 
experiments  and  compare  the  odd  numbered  teams  to  the  even  numbered  teams 
within  each  replication  of  the  experiment.   In  addition,  comparisons  will 
be  made  between  the  results  generated  by  each  of  the  replications. 

As  the  reader  may  recall,  the  primary  difference  between  the  odd 
numbered  and  even  numbered  teams,  within  each  replication,  was  that  the 
odd  numbered  teams  solved  the  problem  manually  before  solving  it  with  the 
aid  of  the  IGRAM  System,  while  the  even  numbered  teams  first  solved  it 
with  the  aid  of  the  computer  system.   The  main  difference  between  the  two 
replications  of  the  experiment  was  the  difference  in  the  managerial  experience 
and  education  of  the  two  sets  of  subjects.   As  we  have  already  stated, 
in  general  the  Senior  Executives  (SE)  had  more  management  and  general 
education  than  did  the  Greater  Boston  Executives  (GBE) .   With  these 
differences  in  mind,  let  us  now  examine  the  experimental  data. 


1.   The  Decision  Made 


The  case  that  we  used  for  the  experiments  was  structured  so  that  the 
net  expected  values  of  the  alternatives  were  very  close  together.   This  was 
done  in  order  to  elicit  the  value  judgments  of  the  subjects,  their  utility 


9 
For  a  more  complete  description  of  the  IGRAM  System,  the  reader  is 

referred  to  Seville,  J.,  et.al.,  3. 
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regarding  risk  and  uncertainty  and  any  other  intangible  elements  which 
might  creep  into  their  decision-making  process.   Given  the  above  experi- 
mental bias,  the  results  of  Table  1,  which  represent  the  number  of  changes 
in  decisions  after  the  experiments,  tend  to  substantiate  our  hypothesis  that 
decision  are  affected  by  the  computer.   A  little  further  we  will  look  into 
the  reasons  behind  most  of  these  switches,  but  right  now  let  us  look  at  the 
data  of  Table  1.   In  the  context  of  the  question  asked,  the  decisions  of 
a  subject  were  defined  as  "switched"  if  one  recommended  that  the  company 
discontinue  making  the  product,  while  the  other  decision  recommended  con- 
tinuation of  operations. 

Of  the  sixteen  Senior  Executives  who  used  the  system  and  completed 
the  experiments,  only  two  reported  a  switch  in  their  decision.    In  sharp 
contrast  nine  out  of  twenty  one  Greater  Boston  Executives  switched.   This 
may  lead  us  to  hypothesize  that  the  Senior  Executives  are  possibly  more 
conservative  than  the  relatively  younger  Greater  Boston  Executives,  or  that 
the  latter  were  not  as  mature  and  thorough  in  their  original  analysis  and 
therefore  learned  relatively  more  after  the  first  solution  than  did  the 
Senior  Executives.   In  our  estimation,  it  is  mainly  the  relative  immaturity 
of  the  Greater  Boston  Executives  in  making  high-level  decisions  which  is 
manifested  in  the  results  and  this  is  supported  by  evidence  presented  in 
Tables  3  and  4. 

If  we  were  to  assume  that  all  the  subjects  belong  to  the  same  universe 
then  we  would  expect  to  find  approximately  five   Senior  Executives  switching 
instead  of  two  and  six  Greater  Boston  Executives  instead  of  nine.   Also,  if 
the  "treatments"  were  neutral  we  would  expect  to  find  about  five  switches 


There  were  twenty  Senior  Executives  in  the  program  but  only  sixteen  volunteere< 
for  the  experiment.   The  respective  figures  for  the  Greater  Boston  Executive 
p-rogram  were  25  participants  and  21  volunteers. 

The  figures  are  rounded  to  the  nearest  integer. 
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Table  1 

Differences  Between  Computer-Aided 
and  Manual  Decisions 


Senior  Executives 


Switches 
No  change 


Total 


Odd  Teams 

Even  Teams 

Total 

2 

0 

2 

7 

7 

14 

9 

7 

16 

Greater  Boston  Executives 


Switches 

5 

No  change 

4 

Total 

9 

Grand  Total 

18 

12 
19 


9 

12 
21 
37 


Note:    Odd  Teams  solved  the  case  first  manually  and  then  used  the 

system  while  the  Even  Teams  did  the  opposite. 
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occurring  among  the  odd  teams  and  six  among  the  even  teams  instead  of 
seven  versus  four  respectively. 

While  we  do  not  wish  to  strain  the  limited  amount  of  data  we  have 
for  statistical  significance,  it  is  evident  to  us  that  the  data  suggest 
that  there  exist  differences  between  the  Senior  Executives  and  the  Greater 
Boston  Executives  and  also  between  the  odd  and  the  even  teams. 

In  Table  2  we  present  the  qualitative  decision  made  by  those  who 
switched.   The  data  reveal  that  the  six  of  the  ten  subjects,  for  whom 
complete  information  exists,  chose  to  continue  the  production  of  the 
product  when  they  used  the  system  and  chose  to  discontinue  its  production 
when  they  solved  the  problem  with  manual  methods.   Five  of  these  six 
subjects  belonged  to  the  odd-numbered  teams  which  solved  the  case  manually 
at  first  and  then  used  the  computerized  system. 

In  order  to  ascertain  the  reasons  behind  the  switches  we  asked  the 
subjects  to  tell  us  which  decision  they  preferred  and  to  qualify  the  change 
of  decision.   Out  of  eleven,  ten  provided  information  and  indicated  that 
all  ten  would  stick  to  their  computer-aided  decision.   The  reasons  behind 
the  "switches"  are  listed  in  Table  3. 

At  first  glance  there  appears  to  be  a  discrepancy  between  some  of  the 
evidence  contained  in  Tables  2,  3  and  the  "absolute"  faith  in  the  computerized 
decision  shown  by  those  who  switched.   Especially  in  view  of  the  four 
switches  among  the  even-numbered  team  members  who,  as  the  reader  may  remember, 
solved  the  probelm  manually  before  using  the  computer.   What  the  subjects  are 
actually  saying  is  that  the  data  they  developed  manually  dictated  a  decision 
opposite  to  the  one  they  arrived  at  with  the  aid  of  the  system.   However,  if 
they  had  to  choose  between  the  two  different  solutions,  they  would  stick  to 
their  computer-aided  decision. 
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Table  2 

Differences  Between  Manual  and  Computer-aided 
Decisions  for  those  subjects  who 
Changed  their  Decisions 


Number  Reporting 


Decisions 


Senior  Executives 


Odd  Numbered  Team  Members 


Manual 


Discontinue 


Computer-aided 
Continue 


Greater  Boston  Executives 


Odd  Numbered  Team  Members 


Manual 

Discontinue 
Continue 


Computer-aided 

Continue 
Discontinue 


Even  Numbered  Team  Members 


Computer-aided 
Discontinue 
Continue 


Manual 


Continue 
Discontinue 
No  data  available 
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Table  3 


Reasons  for  Switching  Decisions  and  Occurrences 


Reasons  Provided 


Odd  Team 


Even  Team 


Senior  Executives 


The  subject  made  errors  in 
manual  solution 

The  assumptions  were  not  exactly 
the  same 


Greater  Boston  Executives 


The  subject  did  not  consider  the 
same  costs  for  every  input  re- 
quired in  both  solutions 

The  computer  provided  more  data 


The  subject  made  errors  in  manual 
solution  1 

No  reason  given  2 
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The  results  of  Table  3  further  indicate  that  the  computer  system  forced 
two  of  the  odd  numbered  team  members  within  the  Greater  Boston  Executives 
to  use  only  the  relevant  variable  costs  in  the  analysis  of  the  alternatives. 
These  subjects  had  used  some  fixed  costs  in  their  previous  manual  solutions, 
but  subsequently  the  computer  guided  them  to  the  relevant  differential 
costs.   In  other  words  the  computerized  system  imposed  a  methodological 
discipline.   Three  even-numbered  team  members  did,  however,  indicate  that 
their  computer-aided  decisions  were  based  on  more  data  than  were  their 
manual  decisions.   In  this  respect  we  must  assume  that  the  additional  data 
were  the  graphs  of  the  distributions  of  net  present  values  since  that  was 
the  only  new  information  which  was  generated  by  the  system.   Unless  of 
course  the  availability  of  easy  computational  power  was  mislabeled  as 
"more  data." 

We  must  admit  that  we  expected  more  dramatic  differences  between  the 
two  modes  of  decision  making.   Possibly,  the  limited  range  of  differences 
between  the  manual  and  computer-aided  decisions  can  be  explained  in  part 
by  the  abstraction  of  realism  in  the  experiment.   A  decision  maker  in 
real-life  situation  would  no  doubt  possess  a  great  deal  more  intuition 
and  judgment  than  did  the  experimental  subjects,  and  he  might  therefore  be 
able  to  specify  better  probability  distributions  than  those  devised  by  the 
subjects.   Also,  as  already  explained,  the  expected  values  of  the  two  major 
alternatives  posed  in  the  problem  were  intended  to  be  so  close  together  that 
any  decision  would  have  to  be  made  on  the  basis  of  the  differences  in  the 
shapes  of  the  net  present  value  distributions,  "intangibles"  and  qualitative 
assumptions.   When  proper  manual  methods  were  used,  the  expected  net  present 
values  for  the  two  major  alternatives--continue  or  discontinue--were  almost 
identical.   However,  when  the  subjects  used  the  system  they  tended  to  make 
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optimistic  sales  projections  which  tipped  the  scales  in  favor  of  continuing 
production  of  the  item.   This  fact  made  it  easy  for  the  subjects  to  make 
their  computer-aided  decision  without  having  to  differentiate  between  the 
alternatives  on  the  basis  of  the  shape  of  their  corresponding  distributions 
of  net  present  values.   Both  the  transcript  of  the  subjects'  use  of  the 
computerized  systems  as  well  as  the  data  of  Tables  tends  to  substantiate 

this  explanation.   As  can  be  seen  in  Table  2  of  the  seven  odd-numbered  team 

12 
members   who  switched,  only  one  decided  to  discontinue  with  the  other  six 


deciding  to  stay  with  the  product. 


2.   The  Decision  Making  Process 

(a)    The  hypotheses  that  subjects  would  rank  risk  higher  as  an  influence 
on  their  computer-aided  decision  than  on  their  manual  decision  received 
mixed  support.   The  Senior  Executives  responses  supported  the  hypothesis 
while  the  Greater  Boston  Executives  responses  did  not  support  it.   In  addition 
to  a  possibility  that  this  hypothesis  is  not  valid  there  may  be  two  other 
explanations  of  the  reaction  of  the  Greater  Boston  Executives:   (i)  Being 
less  mature,  as  managers  these  subjects  may  not  have  appreciated  the 
importance  of  risk  in  decision  making  and  (ii)  the  question  attempted  to 
make  a  ranking  distinction  between  risk,  short-term,  and  long-term  profit- 
ability as  criteria  for  managerial  choice.   It  is  quite  probable  that  these 
subjects  were  unable  to  make  such  fine  distinction  especially  since  risk  is 
not  completely  independent  from  short  and  long  term  profitability. 


12 

The-  reader  is  reminded  that  the  odd-numbered  teams  attacked  the  problem 

manually  at  first  and  then  worked  with  the  system. 
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Table  4 


Number  of  Alternative  Courses  of  Action  Examined 


SE 
Computer-aided 


Odd 


Even 


ni=5.56      m=3.43 
v=2.25      v=0.858 
n  =  9       n  =  7 


Manual 


Odd 


m=5 . 00 


Even 

m=2.86' 
3 


v=6.29       v=0.8 
n  =  8        n  =  7 


Both 


Odd 
m=5.34^ 
v=4.00- 
n  =  17 


Even 
m=3.14^ 
v=0.508- 
n  =  14 


GBE 
Computer-aided 
Odd        Even 
n}=3.67      m=2.66 
v=6.00''     v=0.427'' 
n  =  9       n  =  12 


Manual 


Odd 


Even 


m=3.70  m=2.50 

9  9 

v=5.34  v=0.455 

n  =  9  n  =  12 


Both 


Odd 
m=3.68 
v=5.34 
n  =  18 


10 


11 


Even 


2.58 


10 


v=0.417 
n  =  24 
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Where  m=mean  of  the  sample j  v=variance  of  the  sample,"  and  n=sample  size 


1.  T-test  on  difference  means  is  significant  at  0.1%  level. 

2.  T-test  on  difference  between  means  is  significant  at  5%  level. 

3.  F-test  on  ratio  of  variances  is  significant  at  2. 57=  level. 

4.  T-test  on  the  differences  between  the  means  is  significant  at  the  0.1%  level, 

5.  F-test  on  the  ratio  of  the  variances  is  significant  at  the  1%  level. 
6+8.  Difference  is  significant  at  the  10%  level. 

7+9+11.  Ratio  of  variances  is  significant  at  the  1%,  level. 

10.  Difference  is  significant  at  the  5%  level. 
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(b)   The  results  in  both  replications  only  weakly  supported  the 
hypothesis  that  decision  makers  would  examine  more  alternative  courses 
of  action  when  using  the  system  than  when  using  manual  methods.   Table  4 
contains  the  results  of  the  experiments. 

As  can  be  seen  in  Table  4,  the  only  case  where  a  set  of  teams  re- 
ported examining  fewer  alternatives  when  using  the  computer  system  than 
when  using  manual  methods  (3.67  versus  3.70)  was  the  odd-numbered  GBE  teams. 
All  other  teams  reported  an  increase  although  in  no  case  the  mean 
differences  were  statistically  significant.   The  startling  result  revealed 
by  these  data  is  that  the  odd-numbered  team  members  in  every  instance, 
examined  significantly  more  alternatives  than  did  the  even-numbered  team 
members.   In  the  case  of  the  Senior  Executives,  the  odd-numbered  teams 
examined  5.56  computer-aided  alternatives  versus  5.00  manual  alternatives 
while  the  even-numbered  teams  examined  3.43  and  2.86  alternatives  respectively. 
The  overall  average  number  of  alternatives  examined  was  5.34  for  the  odd- 
numbered  teams  versus  3.14  for  the  even-numbered.   These  data  are 
statistically  significant  at  the  0.17c,  level.   Similar  results  are  revealed 
by  the  Greater  Boston  Executives  experiments  although  not  as  strongly. 
Another  startling  observation  is  the  difference  in  the  variances  between  the 
odd  and  the  even-numbered  teams,  which  differences  are  again  statistically 
significant  at  the  0.1%  level.   The  fact  that  the  odd-numbered  teams  solved 
the  problem  manually  first,  seems  to  have  had  an  important  influence  upon 
their  decision  on  how  many  alternatives  to  examine. 

A  possible  explanation  of  the  results  of  Table  4,  is  that  the  method 
initially  used  by  a  decision-maker  sets  the  style  he  will  use  in  subsequent 
decision  making  including  the  general  number  of  alternatives  he  will  examine 
before  arriving  at  a  decision.   When  the  odd-numbered  teams  first  approached 
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the  problem  manually,  they  were  possibly  not  very  confident  in  their  manual 
methods  of  considering  risk  and  uncertainty.   Therefore,  they  examined 
many  different  alternative  courses  of  action  before  they  made  their  final 
decisions.   The  even-numbered  teams,  however,  first  solved  the  case  with 
the  aid  of  the  computerized  system.   They  possibly  had  a  high  degree  of 
confidence  in  the  way  the  computer  system  handled  risk,  so  they  did  not  feel 
the  need  to  examine  many  alternatives.   The  high  level  of  confidence  of  the 
even  numbered  teams,  therefore,  may  have  led  them  to  a  strange  sense  of 
security  which  resulted  in  their  examining  fewer  alternatives,  on  average, 
than  did  the  odd-numbered  teams.  We  presented  the  results  to  the  participants 
without  expressing  our  views  as  to  the  cause  and  they  provided  the  same 
explanation. 

Hence,  the  method  first  used  by  the  decision-maker  sets  the  style  he 
will  use  in  subsequent  decision-making.   Given  their  respective  styles ,  as 
reflected  by  the  number  of  alternatives  they  considered,  three  of  the  four 
groups  did,  however,  examine  more  alternatives  when  they  used  the  system 
than  when  they  used  manual  methods.  We  must  stress  again,  however,  that  for 
each  group,  the  mean  differences  between  the  number  of  alternatives  examined 
(computer-aided  versus  manual)  were  not  statistically  significant,  contrary 
to  widely  held  notions. 

(c)   In  accordance  with  out  previously  stated  hypothesis,  the  subjects 
in  both  replications  reported  that  they  had  more  confidence  in  their  computer- 
aided  decision  than  they  had  in  their  manual  decision.   This  was  reported 

13 
in  spite  of  the  fact  that  only  11  of  37  subjects  changed  their  decisions. 


13 

As  we  have  already  stressed,  the  subjects  may  have  changed  decisions  or 

not  simply  on  the  basis  of  the  output  data  of  the  method  used.   However,  even 
those  who  switched  decisions  after  they  attacked  the  case  manually  still  had 
more  confidence  in  the  computer-aided  decision. 


-tv"    :'     ;o 
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So,  even  in  the  cases  where  the  manual  and  the  computer-aided  decisions  were 
the  same,  (in  fact  even  in  the  case  where  the  subsequent  manual  method 
dictated  a  switch),  the  subjects  had  more  confidence  in  their  computer- 
aided  decision.   The  causes  for  the  higher  degree  of  confidence  in  the 
computer-aided  decisions  were  traced  to: 

(1)  The  subjects  confidence  in  the  "risk  analysis  method"  of 
handling  uncertainty. 

(2)  The  additional  information  the  system  provided  in  the  form 
of  graphs  of  the  distributions  of  net  present  values. 

(3)  The  structure  imposed  on  the  problem  by  the  system. 

(4)  The  ease  with  which  the  subjects  understood  the  graphical 
information. 

(5)  The  subjects  ability  to  perform  sensitivity  tests,  that  is, 
ask  "what  if"  questions  of  the  system. 

(6)  The  "fact  that  the  system  was  designed  by  experts  who  must  have 
for  sure  incorporated  in  it  the  latest  management  science 
techniques ." 

The  results  under  (b)  and  (c)  above  may  also  indicate  another  phenomenon 

14 
which  Schroder  has  observed  in  his  experiments   .   He  found  that  after  a 

certain  degree  of  complexity  in  the  environment  the  subjects  tend  to  process 

less  information  and  yet  feel  more  confident  in  their  decision.   So  it  could 

be  that  trying  fewer  alternatives,  as  did  the  even-numbered  teams,  was  a 

manifestation  of  an  inability  to  cope  with  the  complexity  of  the  computerized 

case  which  resulted  in  a  bias  toward  a  lower  level  of  information  processing. 

These  are  among  some  challenging  hypotheses  which  we  intend  to  explore  in 

the  future. 


14 

Related  to  one  of  the  authors. 
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IX.   Implications  of  Planning  Models  for  Management 

Based  on  the  results  of  the  experimental  use  of  the  IGRAM  System, 
we  feel  that  planning  models,  such  as  the  IGRAM  System,  hold  an  exciting 
promise  for  management  in  the  future.   The  following  are  some  of  the  uses 
which  we  foresee: 

1)   As  a  central  tool.   The  IGElAM  System  requires  the  user  to  input 
his  subjective  estimates  of  future  states  of  nature,  and  decisions  are  made 
based  on  the  information  generated  from  those  estimates.   All  of  this 
information  can  be  stored  in  machine  readable  form  and  can  be  quickly 
accessed  by  the  user.   It  seems  natural  that  managers  will  want  to  review 
the  progress  of  projects  which  were  approved  with  the  aid  of  the  IGRAM  System 
or  one  similar  to  it.   These  reviews  could  be  made  very  easily  by  examining 
the  original  assumptions  and  estimates  and  comparing  them  with  later 
appraisals  of  the  situation.   Current  data  could  be  periodically  introduced 
into  the  system  to  facilitate  such  reviews.   Quick,  convenient,  real-time, 
computer-aided  reviews  would  help  the  decision-maker  detect  problems — 
assumptions  which  are  not  coming  true  or  critical  estimates  which  are  off 
the  mark--and  use  the  Monte  Carlo  simulation  technique  to  project  the  con- 
sequences of  these  problems  and  evaluate  alternative  solutions  to  the  problem. 
As  the  managers  use  the  system  to  help  control  the  project,  they  would  also 
learn  more  and  more  about  the  nature  of  the  variables  which  are  key  to  the 
project,  and  eventually  develop  a  diagnostic  system  rather  than  do  post  mortems . 

2)   As  an  educational  tool.   A  system,  such  as  IGRAM,  could  be  used 
to  teach  new  managers  the  procedures  used  in  making  decisions  in  the  past-- 
the  elements  considered  the  assumptions  and  estimate  made--and  the  process 
through  which  they  should  progress  in  making  future  decisions.   The  use  of 
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the  system  in  this  manner  will  undoubtedly  reveal  the  weaknesses  of  past 
decisions  as  well  as  their  strength.   The  use  of  such  a  system  appears  to  be 
a  much  more  efficient  method  of  training  managers  than  any  method  in  use  today. 

3)  To  improve  consistency  in  decision  making.   The  structure  and 
discipline  imposed  on  problem  solving  should  produce  more  consistent  and 
efficient  decision  making.   Within  a  firm,  it  would  be  possible  to  have 

the  specialists  in  making  certain  types  of  decisions  develop  their  own  models, 
which  models  in  turn  can  be  used  by  other  managers  within  the  firm  whenever 
they  are  faced  with  one  of  those  types  of  decisions.  Another  advantage 
such  a  system  would  provide  is  that  of  constituting  a  permanent  memory. 

Experience  would  not  be  lost  when  a  "good"  manager  leaves  the  firm.   His 
decision-making  technique  would  be  left  behind  for  others  to  study  and  learn. 
Also  the  "weeding  out"  of  obsolete  methods,  through  automatic  updating  of  planning 
models,  would  eliminate  a  lot  of  the  inefficiencies  we  find  in  practice  today. 

4)  To  improve  communications.   With  a  system  such  as  IGRAM,  qualified 
people  can  specify  the  structure  of  the  problem  and  then  experts  from  the 
various  relevant  areas  can  enter  their  estimates  of  the  variables  related 

to  their  area  of  expertise.  Methods  could  even  be  devised  to  reconcile  the 
differences  between  the  "expert  opinions"  of  several  experts  on  a  single 
variable  in  those  cases  where  there  is  disagreement. 

5)  A  tool  for  consensus  decision-making.  If  we  go  a  step  beyond  the 
resolution  of  differences  between  expert  opinions  we  can  see  how  such  a  system 
could  facilitate  the  resolution  of  differences  between  recommended  courses 

of  action  in  cases  where  there  are  many  decision  makers.   Since  all  inputs 
to  the  system  must  be  explicit,  any  differences  of  opinion  may  be  traced 
to  individual  inputs,  to  the  system,  or  to  individually  held  values.   In 
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either  case,  the  use  of  the  system  can  help  to  isolate  the  cause  of  the 
disagreement  and  hopefully  aid  in  its  resolution.  We  see  interactive 
systems  with  global  models  as  means  of  increasing  communication  and  under- 
standing of  complex  interrelationships. 

Obviously,  the  surface  of  the  field  of  planning  model  development 
and  use  has  only  been  scratched.  Much  is  yet  to  be  discovered  and  validated. 
The  need  is  for  more  detailed,  documented  experimental  work  using  prototype 
models.   The  sooner  we  discover  the  secrets  of  planning  models,  the  sooner 
managers  will  be  able  to  harvest  the  fruits  of  their  use. 
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